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^^^^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Acconnting  Office 
Washington,  DC  20548 


September  28,  2001 

The  Honorable  Sherwood  L.  Boehlert 
Chairman 

Committee  on  Science 
House  of  Representatives 

The  Honorable  F.  James  Sensenbrenner,  Jr. 

House  of  Representatives 

In  fiscal  year  1992,  the  Department  of  Energy  (DOE)  created  the 
Laboratory  Directed  Research  and  Development  (LDRD)  program,  which 
formalized  a  long-standing  policy  of  allowing  its  multi-program  national 
laboratories  discretion  to  conduct  self-initiated,  independent  research  and 
development  (R&D).  DOE  requires  that  LDRD  work  must  focus  on  the 
advanced  study  of  scientific  or  technical  problems,  experiments  directed 
toward  proving  a  scientific  principle,  or  early  analysis  of  experimental 
facilities  or  devices. 

The  Congress  and  the  scientific  community  have  long  recognized  the  value 
of  allowing  laboratories  to  set  aside  part  of  their  budget  to  explore  new 
opportunities  as  an  important  tool  for  maintaining  scientific  excellence  in 
doe’s  national  laboratory  system.  Generally,  at  each  DOE  laboratory, 
researchers  independently  propose  projects  that  are  judged  by  peer  panels 
and  managers  for  their  potential  scientific  merit.  Only  the  most  promising 
projects  are  to  be  chosen  for  LDRD  funding.  Most  projects  cost  $100,000  to 
$300,000  and  last  2  to  3  years.  The  laboratories  pay  for  these  projects  by 
assessing  their  regular  programs’  budgets  a  maximum  of  6  percent — 
except  for  fiscal  year  2000,  when  the  Congress  limited  the  amount  to  4 
percent — and  putting  that  money  into  a  separate  LDRD  account.  All  nine 
of  doe’s  multi-program  national  laboratories  fund  LDRD  projects.^  While 
these  laboratories  conduct  research  in  several  program  areas,  three  focus 
primarily  on  national  security  issues;  one  focuses  on  environmental  issues; 


^  The  nine  laboratories  are  Argoime  National  Laboratory,  Brookhaven  National  Laboratory, 
Idaho  National  Engineering  and  Environmental  Laboratory,  Lawrence  Berkeley  National 
Laboratory,  Lawrence  Livermore  National  Laboratory,  Los  Alamos  National  Laboratory, 
Oak  Ridge  National  Laboratory,  Pacific  Northwest  National  laboratory,  and  Sandia 
National  Laboratories.  DOE  also  funds  14  generally  smaller  laboratories  that  conduct 
specialized  research  and  generally  do  not  participate  in  the  LDRD  program.  However,  one 
of  those  laboratories — the  Ames  Laboratory — has  a  small  LDRD  program  averaging  about 
$196,000  per  year. 
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and  the  other  five  conduct  a  wide  range  of  science  and  technology 
research. 

Concerned  that  some  laboratories  may  have  been  funding  LDRD  projects 
that  did  not  meet  DOE’s  guidelines,  you  asked  us  to 

•  determine  how  much  DOE’s  multi-program  national  laboratories  have 
spent  on  LDRD  projects  since  fiscal  year  1992  (when  the  LDRD 
program  was  created), 

•  evaluate  whether  LDRD  projects  meet  DOE’s  selection  guidelines,  and 

•  provide  observations  on  how  the  program  might  be  improved. 

To  address  these  objectives,  we  reviewed  program-related  information  and 
annual  reports,  budgets,  and  procedures  for  selecting  LDRD  projects  at  all 
nine  of  DOE’s  multi-program  national  laboratories.  At  five  of  those 
laboratories,  representing  83  percent  of  all  LDRD  funding,  we  examined 
the  selection  and  review  processes  that  each  used  to  select  LDRD  projects 
to  determine  if  internal  controls  were  in  place  that  would  reasonably 
ensure  compliance  with  DOE’s  LDRD  project-selection  guidelines.  In 
addition,  we  examined  oversight  practices  used  by  DOE’s  headquarters 
and  field  offices  for  the  LDRD  program.  We  also  randomly  selected  five 
projects  from  each  laboratory  for  detailed  analysis  and  evaluation.  For 
each  project  selected,  we  interviewed  researchers  and  managers  to 
determine  how  their  projects  met  the  guidelines.  Because  of  our  coverage, 
we  believe  our  findings  reflect  the  condition  of  the  review  and  selection 
processes  for  the  vast  majority  of  DOE’s  LDRD  program.  Our  examination 
of  the  randomly  selected  projects  from  the  five  laboratories  provides 
additional  confidence,  at  those  laboratories,  in  the  results  of  the  internal 
control  tests.  Appendix  1  describes  our  scope  and  methodology  in  more 
detail. 


Results  in  Brief 


Since  fiscal  year  1992,  DOE’s  multi-program  national  laboratories  have 
spent  over  $2  billion  (about  $233  million  annually)  on  LDRD  projects. 
DOE’s  three  largest  multi-program  national  laboratories — Lawrence 
Livermore  in  California,  and  Los  Alamos  and  Sandia  in  New  Mexico — 
account  for  nearly  three-quarters  of  laboratory- wide  LDRD  spending. 
These  three  laboratories  concentrate  on  national  security  issues  and,  in 
recent  years,  have  spent  nearly  the  maximum  amount  authorized  by  the 
Congress  for  LDRD  projects — not  more  than  6  percent  of  their  budgets, 
except  for  fiscal  year  2000,  when  the  Congress  limited  the  amount  to 
4  percent.  During  the  course  of  our  review,  DOE  and  laboratory  officials 
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told  us  that  they  believe  that  the  ability  to  offer  innovative  science  work 
through  the  LDRD  program  helps  attract  new  scientists  who  can 
eventually  perform  national  security  research  work.  Thus,  these  officials 
view  LDRD  projects — and  the  scientists  they  attract — as  vital  for  national 
security  in  the  long  term.  DOE’s  other  laboratories  generally  spend  less 
than  4  percent  of  their  budgets  on  LDRD  projects,  and  officials  at  those 
laboratories  also  believe  that  LDRD  projects  are  an  important  way  to 
attract  and  maintain  scientific  talent  and  expertise. 

All  LDRD  projects  we  reviewed  at  the  five  laboratories  we  visited  met 
DOE’s  guidelines  for  selection.  In  addition,  each  of  the  five  laboratories 
had  created  the  internal  controls  necessary  to  reasonably  ensure 
compliance  with  DOE’s  guidelines.  The  key  controls  in  place  included 
using  DOE’s  guidelines  to  control  and  conduct  the  project-selection 
process,  using  individuals  in  the  review  and  selection  process  with  the 
appropriate  skills  and  knowledge  to  evaluate  the  proposed  projects, 
substantially  segregating  duties  among  individuals  to  help  ensure  that  no 
one  individual  is  likely  to  control  the  project-selection  decision  in  a  way 
that  will  violate  LDRD’s  guidelines,  and  ensuring  appropriate  DOE 
oversight  and  review  of  the  results  of  the  process. 

The  LDRD  program  could  improve  its  performance  reporting.  Each 
laboratory  issues  annual  LDRD  reports  that  contain,  among  other  things, 
such  performance  indicators  as  the  numbers  of  patents  obtained, 
publications,  copyrights,  awards,  and  relevance  of  the  research  to  DOE’s 
missions.  However,  while  these  indicators  are  among  those  generally 
accepted  by  the  R&D  community  as  valid,  the  laboratories  do  not  use  a 
common  set  of  performance  indicators  in  their  annual  reports.  Some 
laboratories  report  more  than  a  dozen  measures,  while  others  report  five. 
Additionally,  the  reports  present  performance  information  in  varying 
formats,  making  it  difficult  to  focus  on  the  most  relevant  performance 
information.  As  a  result,  DOE  managers  and  the  Congress  lack  consistent 
performance  information  that  is  needed  to  readily  evaluate  the  overall 
value  of  the  LDRD  program.  Laboratory  managers  told  us  there  is  no 
consensus  on  which  performance  indicators  to  use  when  reporting  the 
results  of  their  LDRD  projects  nor  is  there  an  agreed-upon  reporting 
format.  This  report  makes  a  recommendation  to  the  Secretary  of  Energy  to 
improve  performance  information  reporting  for  the  LDRD  program. 

DOE  agreed  with  our  findings,  conclusions,  and  recommendation  and 
provided  clarifying  comments  which  we  incorporated,  as  appropriate,  in 
this  report. 
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Background 


The  Congress  has  long  recognized  the  value  of  allowing  laboratories  to 
conduct  a  certain  amount  of  discretionary  research.  The  current  LDRD 
program  grew  out  of  legislation  enacted  in  1977  that  authorized  the  use  of 
a  reasonable  amount  of  laboratory  funds  to  conduct  employee-suggested, 
research  and  development  (R&D)  projects  selected  at  the  discretion  of  the 
laboratory  directors.^  DOE’s  implementation  of  its  authority  to  conduct 
discretionary  research  evolved  over  the  years.  For  example, 

•  in  1983,  DOE  Order  5000.  lA  formally  established  a  discretionary  R&D 
program  called  Exploratory  Research  and  Development  (ER&D); 

•  in  1992,  DOE  Order  5000. 4A  established  the  current  LDRD  program, 
which  includes  the  previously  defined  ER&D  program  and  other 
discretionary  work,  and  further  memorialized  its  long-standing  policy 
of  allowing  its  multi-program  national  laboratories  the  discretion  to 
conduct  self-initiated,  independent  R&D;  and 

•  in  1997,  DOE  revised  its  LDRD  program  direction  in  DOE  Order  413.2 
providing  clearer  guidance  on  how  LDRD  funds  may  and  may  not  be 
used. 

Each  of  DOE’s  nine  multi-program  national  laboratories  has  an  LDRD 
program.  Funding  for  LDRD  projects  comes  from  existing  program 
budgets.  Historically,  this  has  been  accomplished  by  allowing  each 
laboratory  to  assess  its  program’s  budgets  at  a  set  rate  of  up  to  6  percent 
and  accumulate  that  money  into  an  overhead  account  for  its  LDRD 
program.^ 

DOE’s  field  offices  oversee  each  laboratory’s  LDRD  program  by  approving 
the  laboratory’s  spending  plans  and  making  sure  that  projects  comply  with 
guidelines.  DOE  also  approves  each  laboratory’s  processes  and 


^  DOE  must  specifically  approve  the  laboratories’  use  of  these  funds.  Legislation 
authorizing  DOE  to  fund  the  national  laboratories’  discretionary  R&D  activities  is  listed  in 
section  6  of  DOE  Order  413.2A,  and  includes  general  authority  for  laboratory  R&D 
activities  in  the  Atomic  Energy  Act  of  1954,  as  amended,  42  U.S.C.  2051,  and  specific 
authority  for  LDRD  programs  in  the  National  Defense  Authorization  Act  for  Fiscal  Year 
1991  (P.  L.  No.  101-510,  section  3132,  42  U.S.C.  7257a). 

^  For  fiscal  year  2000  only,  the  Congress  changed  the  percentage  available  for  LDRD  to  a 
maximum  of  4  percent  and  prohibited  the  use  of  environmental  management  program 
funds  for  LDRD  (Energy  and  Water  Development  Appropriations  Act  of  Fiscal  Year  2000, 
P.  L.  106-60,  Sect.  308). 
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procedures  for  selecting,  reviewing,  and  tracking  LDRD  projects  and 
requires  annual  reports  from  each  laboratory.  DOE  conducts  periodic 
reviews  of  the  laboratories’  management  that  encompass  the  LDRD 
program. 


Laboratories  Have 
Spent  Over  $2  Billion 
on  LDRD  Projects 
Since  Fiscal  Year  1992 


doe’s  nine  multi-program  laboratories  have  invested  over  $2  billion  on 
LDRD  projects  since  1992,  when  the  LDRD  program  was  created.'*  DOE’s 
three  large  defense  laboratories  account  for  a  majority  of  all  LDRD 
spending.  Most  LDRD  funding  is  invested  in  research  supporting  the 
laboratories’  strategic  plans  and  maintaining  the  skills  and  competencies 
necessary  to  carry  out  laboratory  missions.  In  addition,  laboratory 
managers  told  us  they  believe  that  LDRD  projects  help  to  attract  new 
scientists  and  encourage  others  to  explore  cutting-edge  science  projects  in 
order  to  maintain  the  “vitality”  of  the  laboratories.  The  managers  believe 
that  LDRD  projects  also  help  to  identify  new  mission  areas  consistent  with 
DOE’s  overall  mission. 


As  shown  in  table  1,  DOE’s  nine  multi-program  national  laboratories  have 
spent  over  $2  billion  on  LDRD  projects  since  fiscal  year  1992. 


Table  1:  LDRD  Spending  of  DOE’s  Multi-Program  National  Laboratories 


Dollars  In  millions 


Fiscal  year 


National  laboratory 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

Total 

Sandia 

$38.1 

$50.7 

$60.2 

$67.9 

$68.6 

$71.1 

$76.5 

$79.2 

$50.9 

$563.2 

Los  Alamos 

59.3 

63.0 

63.0 

62.1 

54.9 

62.6 

65.0 

71.1 

47.0 

548.0 

Livermore 

40.9 

56.6 

50.3 

50.9 

50.4 

52.5 

55.6 

57.8 

34.5 

449.5 

Argonne 

8.9 

9.6 

11.9 

13.6 

13.4 

14.3 

14.2 

18.1 

17.5 

121.5 

Pacific  Northwest 

12.0 

11.8 

13.7 

15.0 

13.3 

12.9 

11.9 

13.0 

12.4 

116.0 

Oak  Ridge 

8.9 

9.4 

9.8 

12.5 

13.3 

14.6 

14.3 

13.7 

14.6 

111.1 

Idaho 

5.9 

6.1 

10.2 

16.7 

14.4 

11.7 

10.0 

12.5 

4.8 

92.3 

Berkeley 

4.7 

5.0 

5.2 

6.4 

6.8 

9.1 

10.2 

10.6 

10.0 

68.0 

Brookhaven 

1.9 

2.0 

2.3 

2.5 

3.1 

3.5 

2.6 

4.6 

5.6 

28.1 

Total 

$180.6 

$214.2 

$226.6 

$247.6 

$238.2 

$252.3 

$260.3 

$280.6 

$197.3 

$2,097.7 

“*  Prior  to  this  date  laboratories  relied  on  other  programs  to  fund  their  most  innovative 
research. 
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doe’s  three  largest  multi-program  national  laboratories — Lawrence 
Livermore  in  California,  and  Los  Alamos  and  Sandia  in  New  Mexico — 
account  for  nearly  three-quarters  of  all  LDRD  spending.  These  laboratories 
concentrate  on  national  security  issues  and  in  recent  years  spent  near  the 
maximum  amount  authorized  by  the  Congress  for  LDRD  projects — no 
more  than  6  percent  of  their  budgets,  except  for  fiscal  year  2000  when  the 
Congress  limited  the  amount  to  4  percent.  By  contrast,  DOE’s  other 
laboratories  generally  spend  less  than  4  percent  of  their  budgets  on  LDRD 
projects.  (See  table  2.) 


Table  2:  Percentage  of  Budgets  Spent  on  LDRD  for  DOE’s  Multi-Program  National  Laboratories 


Fiscal  year 


National  laboratory 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

Sandia 

3.00 

4.00 

4.60 

5.10 

5.20 

5.42 

5.74 

5.80 

3.75 

Los  Alamos 

5.77 

5.73 

5.86 

5.67 

5.32 

5.66 

5.47 

5.65 

3.81 

Livermore 

4.00 

5.95 

5.86 

5.97 

5.73 

5.98 

5.36 

5.42 

3.31 

Argonne 

2.23 

2.24 

2.51 

2.76 

2.81 

3.11 

3.09 

3.90 

3.58 

Pacific  Northwest 

3.20 

2.84 

3.02 

3.11 

3.17 

2.61 

2.75 

2.78 

2.67 

Oak  Ridge 

1.83 

1.85 

1.68 

2.02 

2.22 

2.48 

2.30 

2.20 

2.52 

Idaho 

1.12 

1.19 

1.90 

2.08 

2.13 

1.89 

1.84 

1.95 

0.70 

Berkeley 

1.80 

1.80 

1.91 

2.20 

2.00 

2.94 

3.10 

3.10 

2.70 

Brookhaven 

0.67 

0.72 

0.81 

0.84 

1.03 

1.13 

0.86 

1.32 

1.39 

Each  of  the  nine  multi-program  national  laboratories  established  separate 
but  similar  LDRD  categories  of  funding,  using  these  as  guides  to  selecting 
proposals.  The  number  of  categories  ranged  from  one  to  five.  In  most 
laboratories,  the  largest  category  contained  projects  that  aligned  most 
closely  with  the  laboratory’s  strategic  missions,  such  as  the  principal 
missions  of  national  security  at  the  defense  laboratories  and  fundamental 
science  at  the  Lawrence  Berkeley  National  Laboratory.  These  types  of 
LDRD  projects  tended  to  be  larger  and  were  expected  to  have  nearer-term 
results.  The  second  largest  category  was  generally  directed  at  building 
scientists’  skills  and  strengthening  laboratory  competencies.  Generally,  the 
laboratories  target  the  smallest  amount  of  funding  to  projects  that  are  the 
highest  risk  and  most  cutting-edge  as  shown  in  the  following  examples: 

•  The  Lawrence  Livermore  National  Laboratory  has  three  main 

categories  of  funding.  Strategic  Initiatives  projects  represent  27  percent 
of  all  LDRD  funds,  focus  on  research  addressing  national  needs  in 
support  of  the  laboratory’s  strategic  vision,  and  are  larger 
multidisciplinary  projects.  Exploratory  Research  projects  received  67 
percent  of  the  funds,  support  the  strategic  vision  and  competencies 
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building  of  programs  and  directorates  across  the  laboratory,  and  are 
smaller  than  the  Strategic  Initiatives  projects.  Laboratory- wide  projects 
received  about  6  percent  of  the  funds,  are  designed  to  encourage 
creativity  of  individual  scientists  in  the  pursuit  of  innovative  research, 
and  are  funded  at  a  maximum  of  $180,000.  A  category  of  funding  that 
receives  less  than  1  percent  of  the  laboratory’s  LDRD  funds — 
Definition  and  Feasibility  Study  projects — provides  the  seeds  for  new 
research  ideas  and  are  usually  funded  for  less  than  6  months  and 
$50,000. 

•  The  Los  Alamos  National  Laboratory  has  two  categories  of  LDRD 
projects.  Directed  Research  projects  received  about  two-thirds  of  the 
funds,  support  the  laboratory’s  strategic  plan,  are  typically 
multidisciplinary,  and  generally  cost  $1  million  or  more.  Exploratory 
Research  projects  received  about  one-third  of  LDRD  funds,  are  usually 
smaller  and  the  most  innovative,  and  generally  cost  $250,000  or  less. 

•  The  Pacific  Northwest  National  Laboratory  has  three  categories  for 
LDRD  projects.  Laboratory-level  projects  received  about  two-thirds  of 
the  laboratory’s  LDRD  funds  and  are  for  projects  that  directly  align 
with  the  laboratory’s  primary  research  areas,  are  generally  multiyear 
and  multidisciplinary,  and  cost  from  $100,000  to  $250,000.  Division- 
level  projects  received  about  one-third  of  the  LDRD  funds,  are  aimed  at 
developing  new  ideas  in  a  particular  mission  area,  have  intermediate 
and  near-term  mission  relevance,  and  cost  from  $80,000  to  $100,000. 
Level  VI  projects,  which  received  a  total  of  about  $500,000  of  the 
laboratory’s  LDRD  budget,  are  intended  to  support  highly  innovative 
ideas,  and  typically  cost  less  than  $60,000  each. 

DOE  and  laboratory  officials  believe  that  the  innovative  nature  of  LDRD 
projects  helps  attract  new  scientists  who  can  contribute  to  maintaining  the 
vitality  of  the  laboratories.  Those  officials  focusing  on  national  security 
issues  believe  that  the  LDRD  program  helps  attract  scientists  who  can 
eventually  perform  national  security  research  work.  They  believe  that 
because  nuclear  weapons  science  is  not  taught  in  colleges  and  must  be 
taught  within  the  defense  laboratories,  LDRD  projects — and  the  scientists 
they  attract — are  vital  for  national  security  in  the  long  term.  For  example, 
postdoctoral  students  represent  a  major  source  of  future  research  staff  at 
the  laboratories,  and  most  of  them  are  hired  to  work  on  LDRD  projects. 
Sixty-two  percent  of  Sandia’s  postdoctoral  staff  hired  between  1996  and 
1999  worked  on  LDRD  projects. 
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LDRD  Projects  Met 
doe’s  Guidelines 


doe’s  Laboratory  Operations  Board,  comprising  internal  managers  and 
external  consultants,  reported,  in  January  2000,  that  LDRD  programs  are 
vital  in  recruiting  and  retaining  the  best  scientific  talent  into  the 
laboratories.^  According  to  the  Board’s  report,  from  1993  through  1998, 

41  percent  of  LDRD-funded  postdoctoral  staff  at  Lawrence  Livermore 
National  Laboratory — a  defense  program  laboratory — were  subsequently 
hired  by  the  laboratory.  Officials  from  nondefense  program  laboratories 
also  told  us  that  LDRD  projects  are  important  for  attracting  and 
maintaining  scientific  talent  in  their  laboratories.  These  laboratories, 
however,  spend  less  on  LDRD  than  defense  program  laboratories  for  a 
number  of  reasons,  including  that  they  conduct  more  basic  science  work 
as  a  primary  mission  within  their  regular  programs. 


All  of  the  randomly  selected  LDRD  projects  we  reviewed  at  the  five 
laboratories  we  visited  met  DOE’s  guidelines  for  selection.  Additionally, 
doe’s  and  the  laboratories’  management  controls  were  adequate  to 
reasonably  ensure  that  approved  projects  would  likely  meet  DOE’s 
project-selection  guidelines.  DOE’s  guidelines  specify  that  LDRD  projects 
must  be  in  the  forefront  of  science  and  technology  and  should  include  at 
least  one  of  the  following: 

•  Advanced  study  of  hypotheses,  concepts,  or  innovative  approaches  to 
scientific  or  technical  problems. 

•  Experimentation  and  analyses  directed  toward  “proof  of  principle”  or 
early  determination  of  the  utility  of  new  scientific  ideas,  technical 
concepts,  or  devices. 

•  Conception  and  preliminary  technical  analyses  of  experimental 
facilities  or  devices. 

In  addition,  DOE’s  guidelines  generally  require  that  LDRD  projects  should 
not  last  longer  than  36  months,  be  supplemented  by  non-LDRD  funds,  be 
used  to  perform  or  supplement  funding  for  DOE’s  program  work,  or  be 
used  to  fund  construction  for  scientific  projects  beyond  the  preliminary 
phase  of  the  research. 


^  See  Review  of  the  Department  of  Energy’s  Laboratory  Directed  Research  and 
Development  Program,  Report  of  the  External  Members  of  the  Laboratory  Operations 
Board  (Jan.  27,  2000). 
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Projects  Met  DOE’s 
Guidelines 


All  LDRD  projects  we  reviewed  met  DOE’s  guidelines.  These  projects  were 
new  projects  that  were  proposed  for  fiscal  year  2000  funding.  Most  of 
these  projects  tested  or  analyzed  a  new  or  untested  concept  and  were 
consistent  with  the  laboratory’s  strategic  missions,  as  shown  in  the 
following  examples: 

•  A  Los  Alamos  project  has  a  goal  of  advancing  the  state  of  fundamental 
simulation  theory  so  that  sophisticated  simulation  tools  can  be 
developed  for  use  in  decision-making  in  complex  national  security 
environments,  such  as  critical  national  infrastructure  analysis  and 
military  engagements.  The  project  involves  developing  complex 
integrated  simulation  tools  that  will  advance  fundamental  research  in 
the  areas  of  mathematical  foundations  of  simulation,  issues  in 
implementing  and  computing  for  large  simulations,  statistical  methods 
for  simulation-based  studies,  and  principles  for  simulation-based 
assisted  reasoning.  The  project’s  results  are  primarily  targeted  to  have 
relevance  in  mobile  communications,  regional  population  mobility  and 
transportation  infrastructure,  electrical  power  distribution  networks 
and  markets,  epidemiological  impacts  on  populations,  and  threat 
identification  and  targeting  in  urban  terrain.  In  the  project’s  first  year, 
among  other  things,  demonstrations  will  focus  on  mobile 
telecommunications,  transportation  systems,  and  epidemiological 
impacts.  The  project  is  being  done  under  Los  Alamos’  Directed 
Research  category  and  supports  the  laboratory’s  strategic  goals  in 
threat  reduction,  high-performance  computing,  and  modeling  and 
simulation.  The  project  was  proposed  for  3  years;  $600,000  was 
approved  for  first-year  funding. 

•  An  Argonne  National  Laboratory  project  is  designed  to  fabricate 
magnetic  wires  of  20  nanometers  (a  nanometer  is  one-billionth  of  a 
meter)  down  to  atom  scale  and  study  their  static  and  dynamic  magnetic 
properties.  This  project  complements  Argonne’s  mission  in  the 
materials  science  area  and  could  help  define  a  new  research  direction 
for  the  laboratory.  The  ultimate  goal  is  to  create  a  new  generation  of 
miniaturization  in  electronics,  including  memories,  transistors,  logic 
elements,  and  sensors.  The  physical  size  of  a  magnetic  system  may 
affect  its  magnetic  properties;  this  project  proposes  to  study  this 
phenomenon  and  make  major  inroads  in  understanding  the 
fundamental  issues  of  low-dimensional  magnetic  systems.  These  issues 
require  a  basic  understanding  of  magnetic  thin  films  and  multilayers 
used  in  computing  today  as  well  as  a  deeper  understanding  of  one¬ 
dimensional  nanotechnology  and  synthesis  of  materials  in  this 
environment.  The  project  managers  plan  to  develop  samples 
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unprecedented  in  the  study  of  lower-dimensional  systems  to  better 
explore  fundamental  questions  about  the  next-generation  magnetism 
research.  This  project  is  being  done  under  Argonne’s  category  of 
funding  for  more  innovative  projects — the  Director’s  Competitive 
Grants  Program.  The  project  was  proposed  for  2  years;  $65,000  was 
approved  for  first-year  funding. 


•  A  Sandia  National  Laboratories  project  aims  to  develop  new  scientific 
tools  for  addressing  the  threat  of  biological  terrorism,  which  is 
consistent  with  Sandia’s  national  security  mission.  Currently,  the  ability 
to  initially  detect  people  exposed  to  a  released  agent  relies  on  the 
outward  appearance  of  symptoms,  such  as  lethargy  and  fever.  The  goal 
of  this  proposed  LDRD  project  is  to  show  that  earlier  detection,  based 
on  cellular-level  changes  in  the  body  through  blood  analysis,  could  be 
accomplished.  The  project  also  aims  to  develop  techniques  and  models 
to  detect  and  analyze  infection  without  waiting  for  external  symptoms. 
The  results  could  reduce  disease  detection  time  from  days  to  hours. 

The  development  of  a  rapid,  highly  sensitive  screening  mechanism 
would  also  have  widespread  application  in  the  fight  against  other 
infectious  diseases.  This  9-month  project  costs  $100,000  and  falls  under 
Sandia’s  Development  Reserve  category,  which  is  used  for  urgent 
science  and  technology  needs  or  technical  work  related  to 
development  of  a  new  program. 


Adequate  Management 
Controls  Exist  to 
Reasonably  Ensure 
Compliance  With  LDRD’s 
Guidelines 


DOE  and  laboratory  management  controls  were  adequate  to  reasonably 
ensure  that  projects  approved  would  likely  meet  DOE’s  project-selection 
guidelines.  The  key  controls  in  place  included  using  DOE’s  guidelines  to 
control  and  conduct  the  project-selection  process,  using  individuals  in  the 
review  and  selection  process  with  the  appropriate  skills  and  knowledge  to 
evaluate  the  proposed  projects,  substantially  segregating  duties  among 
individuals  to  help  ensure  that  no  one  individual  is  likely  to  control  the 
project-selection  decision  in  a  way  that  will  violate  LDRD’s  guidelines,  and 
ensuring  appropriate  DOE  oversight  and  review  of  the  results  of  the 
process. 


All  laboratories  used  DOE’s  LDRD  Order  413.2  as  the  primary  guidance  to 
review  and  select  projects.  Individuals  involved  in  the  review  and  selection 
of  the  projects  had  the  requisite  background  and  experience  to  provide 
credible  review.  Those  individuals  had  wide-ranging  scientific 
backgrounds — usually  a  Ph.D.  in  scientific  research  and  practical 
experience  in  basic  scientific  research.  When  the  subject  matter  of  a 
project  proposal  was  outside  the  knowledge  base  of  the  review  team,  the 
laboratories  generally  contracted  with  outside  experts  to  provide  reviews 
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and  recommendations  on  the  merits  of  that  proposal.  In  general,  each 
laboratory  established  review  panels  comprising  individuals  from  across 
the  laboratory,  which  provided  for  diverse  opinions  to  ensure  that  various 
points  of  view  were  brought  to  bear  on  the  selection  decision.  In  general, 
the  review  panels  consisted  of  managers  from  directorates  having 
knowledge  in  the  project  subject  area,  other  subject  matter  experts,  and 
managers  from  the  LDRD  program.  Finally,  DOE’s  field  offices,  which  are 
responsible  for  overseeing  each  laboratory,  annually  review  the 
laboratories’  recommendations  for  projects  to  be  funded  and  forward 
recommendations  to  headquarters  for  approval.  While  DOE’s  reviews  of 
proposed  projects  have  resulted  in  clarifications  and  minor  revisions  in 
the  proposals’  documentation,  those  reviews  have  rarely  resulted  in  not 
funding  proposed  projects. 


All  laboratories  we  reviewed  have  separate  and  somewhat  different  review 
and  selection  processes  linked  to  their  distinct  categories  of  funding  for 
LDRD  projects,  but  key  elements  of  these  processes  are  very  similar.  For 
example,  the  laboratories  we  visited  initiate  their  annual  LDRD  selection 
process  by  asking  research  staff  to  propose  potential  projects,  called  “calls 
for  proposals.”  These  calls  ask  for  proposals  that  generally  fit  into  a 
particular  category  of  funding  in  the  LDRD  program.  Reviewers  for  the 
individual  categories  of  funding  review  those  proposals  and  either  reach 
consensus  or  vote  outright  on  where  each  proposal  should  be  ranked  in 
terms  of  recommending  it  for  funding.  That  recommendation  is  then 
generally  given  to  the  laboratory  director,  who  selects  the  projects  to  be 
funded.  The  projects  recommended  for  funding  are  given  to  DOE’s  field 
offices  for  review  and  comment  and  ultimately  forwarded  to  DOE’s 
headquarters  for  approval. 


Opportunities  Exist  to 
Improve  LDRD’s 
Performance 
Reporting 


The  LDRD  program  could  improve  its  performance  reporting.  Each 
laboratory  issues  an  annual  LDRD  report  that  includes  performance 
reporting,  but  those  reports  do  not  use  a  common  set  of  performance 
indicators.®  Additionally,  the  reports  present  performance  information  in 
varying  formats,  making  it  difficult  to  focus  on  the  most  relevant 
performance  information.  Laboratory  managers  told  us  there  is  no 
consensus  on  which  performance  indicators  to  use  when  reporting  the 
results  of  their  LDRD  projects,  nor  is  there  an  agreed  upon  reporting 


®  DOE  issues  a  separate,  annual,  legislatively  mandated  LDRD  report  on  how  funds  spent 
on  the  program  support  the  national  security  mission.  These  reports  cover  the  three 
defense  program  laboratories.  The  reports  are  in  addition  to  the  LDRD  program’s  annual 
reports  issued  by  each  LDRD  laboratory. 
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format.  While  the  reports  describe  the  accomplishments  of  individual 
laboratories,  taken  together,  the  laboratories’  reports  do  not  provide 
aggregate  performance  information  that  DOE  managers  and  the  Congress 
could  use  to  readily  assess  the  overall  value  of  the  program. 

The  different  performance  indicators  reported  in  each  of  the  laboratories’ 
annual  LDRD  reports  make  it  difficult  to  readily  assess  overall  program 
performance  for  DOE’s  LDRD  program.  Table  3  provides  a  summary  of  the 
performance  information  included  in  the  annual  LDRD  reports  published 
by  the  nine  multi-program  national  laboratories  in  our  review  and 
demonstrates  the  lack  of  imiformity  in  reporting  the  LDRD  program’s 
results  across  the  laboratory  complex. 


Table  3:  Performance  Indicators  Used  in  LORD’S  Annual  Reports  of  DOE’s  Multi-Program  National  Laboratories  for  Fiscal 

Year  2000 

Performance  indicators 

Laboratory 

Argonne 

Berkeley 

Brook- 

haven 

Idaho 

Livermore 

Los 

Alamos 

Oak 

Ridge 

Pacific 

Northwest 

Sandia 

Refereed  publications 

X 

s 

X 

X 

X 

X 

All  other  reports  and  publications 

X 

X 

X 

X 

Publications  (without 
differentiation) 

X 

X 

X 

X 

Presentations 

X 

X 

X 

Symposia 

X 

X 

X 

Patent  disclosures 

X 

s 

X 

X 

X 

X 

X 

Patent  applications 

X 

s 

X 

X 

X 

Patents 

X 

X 

X 

s 

s 

X 

X 

X 

Copyrights 

X 

X 

X 

X 

X 

Licensing  agreements 

X 

Students 

X 

s 

X 

X 

X 

X 

Post-doctoral  staff 

X 

s 

X 

s 

X 

X 

X 

Permanent  staff  hired 

X 

s 

s 

s 

X 

X 

Awards 

X 

s 

s 

X 

X 

New,  non-LDRD-funded  projects 

X 

X 

X 

X 

X 

Cooperative  R&D 

X 

X 

X 

Work  for  others  agreements 

X 

X 

X 

Retrospective  study 

X 

Percentage  of  milestones 
completed 

X 

Goals  status 

X 

Hypotheses  status 

X 

Legend: 


X  =  Information  reported  on  individual  projects 
S  =  Information  reported  in  summary  format  for  all  projects 
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In  general,  the  laboratories  maintain  more  detailed  performance 
information  than  they  report  in  their  annual  reports,  but  laboratory 
officials  do  not  agree  on  a  set  of  performance  indicators  that  should  be 
reported  on  for  the  program.  Some  pointed  out  that  there  is  a  significant 
difference  between  different  types  of  publications.  Refereed  publications, 
for  example,  must  go  through  an  expert  review  process  before  they  can  be 
published.  Also,  certain  publications  have  higher  levels  of  difficulty  and 
achievement  and,  therefore,  significance.  The  same  issue  surrounds  the 
tabulation  of  awards  as  performance  indicators.  Likewise,  symposia,  as 
well  as  other  potential  measures,  carry  different  degrees  of  significance. 
Many  suggested  that  success  stories  are  the  best  measures  of  a  project’s 
performance,  particularly  for  basic  research  whose  ultimate  value  may  not 
be  evident  for  a  long  time.  Furthermore,  they  told  us  that  projects  viewed 
as  unsuccessful  with  respect  to  their  direct  proposed  goals  might  in  fact 
have  answered  critical  questions  that  paved  the  way  for  major 
breakthroughs  in  science. 

In  addition,  we  found  that  differences  in  how  performance  information  is 
presented  in  the  laboratories’  annual  LDRD  reports  also  make  it  more 
difficult  to  assess  the  overall  value  of  the  program.  As  indicated  in  table  3, 
we  found  that  while  some  laboratories  present  performance  information 
for  individual  projects,  other  laboratories  present  performance 
information  in  a  summary  fashion.  Two  contrasting  performance-reporting 
styles  can  be  found  in  Sandia  National  Laboratories’  and  Lawrence 
Livermore  National  Laboratory’s  annual  LDRD  reports.  Sandia’s  report 
provides  an  appendix  entitled  “Project  Performance  Measures,”  which  lists 
LDRD  projects  and  catalogues  outputs  of  the  projects  using  11  quantitative 
performance  indicators  and  several  qualitative  indicators.  In  contrast, 
Lawrence  Livermore’s  LDRD  report  provides  an  appendix  listing 
publications  resulting  from  individual  LDRD  projects  and  describes — in 
summary  format  rather  than  on  a  project-by-project  basis — several  other 
quantitative  performance  indicators,  including  patents,  awards,  and 
permanent  staff  hired. 

While  the  laboratories’  annual  LDRD  reports  describe  the 
accomplishments  of  individual  laboratories,  taken  together,  the 
laboratories’  reports  do  not  provide  aggregate  performance  information 
that  DOE  managers  and  the  Congress  could  use  to  readily  assess  the 
overall  value  of  the  program.  Aggregate,  more-uniform  performance 
reporting  on  the  LDRD  program  could  aid  DOE  managers,  the  Congress, 
and  others  in  their  oversight  of  the  program. 
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Conclusions 

In  general,  LDRD  project-selection  and  review  processes  in  place  at  DOE’s 
multi-program  national  laboratories  are  adequate  to  reasonably  ensure 
compliance  with  DOE’s  project-selection  guidelines.  Our  review  of 
randomly  selected  LDRD  projects  at  laboratories  found  that  they  met 
DOE’s  guidelines.  However,  our  observations  of  the  performance¬ 
reporting  practices  for  the  LDRD  program  lead  us  to  conclude  that 
performance  reporting  for  the  program  could  improve.  By  reporting 
aggregate,  more-uniform  performance  information  for  the  LDRD  program 
as  a  whole,  DOE  managers  and  the  Congress  could  more  readily  assess  the 
overall  value  of  the  program. 

Recommendation  for 
Executive  Action 

1 - 

To  improve  the  Congress’s  ability  to  make  informed  decisions  on  the  value 
of  the  LDRD  program,  we  recommend  that  the  Secretary  of  Energy 
develop  and  annually  report  aggregate,  more-uniform  performance 
information  for  the  LDRD  program.  This  recommendation  will  require 
DOE’s  National  Nuclear  Security  Administration  and  the  Office  of  Science, 
which  are  both  accountable  for  laboratory  performance,  to  work  together 
and  develop  performance  indicators  that  can  be  used  to  demonstrate 
accomplishments  across  all  the  laboratories. 

Agency  Comments 

1 - 

We  provided  a  draft  of  this  report  to  DOE  for  review  and  comment. 

According  to  representatives  of  the  Office  of  Science  responsible  for  the 
LDRD  program,  DOE  agreed  with  our  findings,  conclusions,  and 
recommendation.  DOE  also  provided  a  number  of  clarifying  comments, 
which  we  incorporated,  as  appropriate,  in  this  report. 

As  arranged  with  your  offices,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  15  days  after  the 
date  of  this  letter.  At  that  time,  we  will  send  copies  to  the  Secretary  of 
Energy  and  the  Director,  Office  of  Management  and  Budget.  We  will  also 
make  copies  available  to  others  on  request. 
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If  you  or  your  staff  have  any  questions  about  this  report,  please  call  me  at 
(202)  612-3841.  Key  contributors  to  this  report  were  Gary  Boss,  Michael 
Gilbert,  and  Dennis  Carroll. 


(Ms.)  Gary  L.  Jones 
Director,  Natural  Resources 
and  Environment 
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Appendix  I;  Scope  and  Methodology 


To  determine  how  much  the  Department  of  Energy’s  (DOE)  multi-program 
national  laboratories  have  spent  on  Laboratory  Directed  Research  and 
Development  (LDRD)  projects  since  1992  (when  the  LDRD  program  was 
created),  we  reviewed  program  information,  including  annual  reports, 
budgets  and  other  financial  information  provided  by  DOE  and  laboratory 
officials  for  the  nine  DOE  multi-program  national  laboratories.  These 
laboratories  are 

•  Argonne  National  Laboratory, 

•  Brookhaven  National  Laboratory, 

•  Idaho  National  Engineering  and  Environmental  Laboratory, 

•  Lawrence  Berkeley  National  Laboratory, 

•  Lawrence  Livermore  National  Laboratory, 

•  Los  Alamos  National  Laboratory, 

•  Oak  Ridge  National  Laboratory, 

•  Pacific  Northwest  National  Laboratory,  and 

•  Sandia  National  Laboratories. 

Although  doe’s  Ames  Laboratory  has  a  LDRD  program,  we  excluded  it 
from  our  review  because  Ames  is  not  a  multi-program  national  laboratory. 

To  determine  if  LDRD  projects  met  DOE’s  selection  guidelines,  we 
reviewed  the  procedures  and  processes  for  selecting  LDRD  projects  at  all 
nine  of  DOE’s  multi-program  national  laboratories.  We  also  tested  the 
internal  controls  for  project  selection  at  five  of  those  laboratories  and  the 
respective  DOE  offices  responsible  for  oversight  of  the  program,  and 
randomly  selected  approved  LDRD  projects  at  those  five  laboratories.^  The 
five  laboratories  were  Argonne  National  Laboratory,  Lawrence  Berkeley 
National  Laboratory,  Lawrence  Livermore  National  Laboratory,  Los 
Alamos  National  Laboratory,  and  Sandia  National  Laboratories.  These 
laboratories  include  three  of  the  largest  multi-program  national 
laboratories  and  represent  83  percent  of  DOE’s  LDRD  expenditures  for  the 
period  reviewed.  We  also  interviewed  DOE’s  field  office  officials 
responsible  for  the  oversight  of  the  program  in  the  Albuquerque,  Chicago, 
Idaho,  and  Oakland  Operations  offices.  In  addition,  we  interviewed 
officials  responsible  for  the  LDRD  program  in  DOE’s  headquarters  Office 


^  In  planning  our  assignment,  of  the  nine  multi-program  national  laboratories,  we  visited  aU 
but  the  Brookhaven  National  Laboratory.  From  all  nine  laboratories  we  obtained 
documentation,  including  program  costs,  processes  and  procedures  used  to  select  projects 
for  funding,  and  annual  reports.  We  also  interviewed  laboratory  officials  at  Brookhaven. 
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of  Science,  Office  of  Defense  Programs,  and  Office  of  Environmental 
Management. 

To  test  the  internal  controls  of  the  program,  we  evaluated  the  processes 
and  procedures  used  to  select  LDRD  projects.  The  internal  control  tests 
were  designed  to  determine  if  adequate  management  control  was  built  into 
the  LDRD  program  to  provide  reasonable  assurance  that  projects 
approved  through  the  program  comply  with  DOE’s  guidelines  for  the 
LDRD  program.  We  performed  the  internal  control  tests  by  examining  the 
processes  and  procedures  to  ensure  that  the  (1)  people  involved  in  the 
selection  of  the  LDRD  projects  used  the  same  guidance  and  selection 
criteria,  (2)  individuals  involved  in  the  selection  of  the  projects  had  the 
appropriate  skills  and  knowledge  to  evaluate  the  proposed  projects, 

(3)  duties  in  the  project-selection  process  were  segregated  substantially 
among  individuals  so  that  no  one  individual  would  be  likely  to  control  the 
project-selection  decision  in  a  way  that  would  violate  LDRD  guidelines, 
and  (4)  DOE  oversight  activities  were  adequate.  To  accomplish  this,  we 
obtained  from  the  respective  DOE  officials  and  laboratory  management 
officials,  documentation  and  interview  information  on  guidance  provided 
to  the  LDRD  project  review  and  selection  personnel  on  how  to  select 
LDRD  projects  for  funding  at  each  laboratory.  We  then  obtained 
documentation  on  the  LDRD  processes  and  procedures  for  reviewing  and 
selecting  projects  for  funding  at  each  of  the  five  laboratories.  This 
information  included  documentation  on  how  proposed  projects  originate 
to  final  selections  or  other  dispositions.  We  obtained  documentation  and 
interview  information  on  which  individuals  participate  in  each  phase  of 
the  process,  their  roles,  and  their  backgrounds. 

Using  random  number  tables,  we  selected  five  projects  from  each  of  the 
five  selected  multi-program  national  laboratories’  projects  approved  for 
funding  for  fiscal  year  2000^ —  a  total  of  25  projects.  Because  each 
laboratory  had  more  than  one  category  of  LDRD  funding  and  to  enable  us 
to  review  projects  within  each  of  those  categories,  we  randomly  selected 
at  least  one  project  from  each  category  of  funding  at  each  laboratory.  We 
determined  if  each  selected  project  met  DOE’s  project-selection  guidelines 
and,  to  complement  our  internal  control  tests,  we  examined  the  elements 
of  the  processes,  qualifications  of  reviewers,  segregation  of  duties,  and 
DOE’s  oversight.  For  each  of  these  projects,  we  reviewed  the  project 


^  At  the  time  of  our  review,  this  was  the  most  recent  period  for  which  LDRD  projects  had 
been  approved. 
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proposal  files,  including  documentation  reflecting  individual  reviewers’ 
recommendations  on  the  disposition  of  each  case.  We  also  interviewed  the 
scientists  who  proposed  each  project  and  the  laboratory  officials 
responsible  for  reviewing  the  projects  for  selection  to  better  understand 
the  technical  nature  of  the  research  and  how  that  research  meets  DOE’s 
guidelines  for  LDRD  projects.  Interviews  with  selection  officials  also 
focused  on  determining  if  individuals  involved  in  the  selection  of  the 
projects  had  the  appropriate  skills  and  knowledge  to  evaluate  the 
proposed  projects  and  if  the  duties  in  the  process  were  segregated  so  that 
no  one  individual  would  be  likely  to  control  the  project-selection  decision 
in  a  way  that  would  violate  LDRD’s  criteria.  We  also  interviewed  DOE 
officials  in  headquarters  and  the  field  offices  involved  in  the  oversight 
process  through  which  the  projects  were  selected.  While  we  cannot 
project  the  results  of  our  analysis  of  LDRD  projects  to  the  universe  of 
those  projects,  our  analysis  provides  a  snapshot  of  how  internal  controls 
were  being  applied,  and  additional  confidence,  at  the  five  selected 
laboratories,  in  the  results  of  our  internal  control  testing  overall. 

To  provide  views  on  how  the  program  might  be  improved,  we  relied  on 
observations  obtained  throughout  the  course  of  our  audit  work. 

We  provided  a  draft  of  this  report  to  DOE  for  review  and  comment. 
According  to  representatives  of  the  Office  of  Science  responsible  for  the 
LDRD  program,  DOE  agreed  with  our  findings,  conclusions,  and 
recommendation.  DOE  also  provided  a  number  of  clarifying  comments, 
which  we  incorporated,  as  appropriate,  in  this  report. 

Our  review  was  performed  from  December  1999  through  September  2001 
in  accordance  with  generally  accepted  government  auditing  standards. 
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